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METHOD AND APPARATUS FOR 
MANAGING STREAMING DATA 



CROSS REFERENCE TO RELATED APPLICATIONS 

5 This application claims priority from provisional patent application 60/157,998 

entitled STREAMING MACHINE-AWARE BINARY AND MULTIMEDIA CONTENT 
FOR MULTIMEDIA SYNCHRONIZATION filed 10/06/99. 

The above referenced application(s) are incorporated herein by reference for all 
purposes. These prior applications, m some parts, may indicate earlier efforts at 
10 describing the invention or describing specific embodiments and examples. The present 
invention is therefore best understood as described herein. 

FIELD OF THE INVENTION 

The invention generally relates to methods and/or devices related to data 

transmission and presentation using a computing device or information appliance. More 
15 particularly, the invention in various aspects relates to multimedia data methods and 
applications and to a versatile file structure suitable for providing multimedia data and 
streaming data over a communication data network. 

BACKGROUND OF THE INVENTION 

Previous works related to aspects of the present discussion include below listed 

20 reference by Blakowski et al. and by the VV\VW Consortium. The Blakowski reference 
provides a survey of research regarding media synchronization. The reference discusses a 
four-layer synchronization reference model. From the lower layer to the upper layer are 
Media Layer, Stream Layer, Object Layer, and Specification Layer. A variety of 
techniques have been published and implemented in the specification area. SMIL is one 

25 of the latest efforts in the industr>' to provide a presentation specification using 
descriptive tagged language, which follows the syntax of XML. 

Existing and popular media container formats for computers include the 
QuickTime from Apple Computer (http://\v\vw. apple. com), ASF/AVl from Microsoft 
(httpV/w-ww. microsoft.com), RealMedia from RealNetworks (http://www.real.com), and 

30 Video Capsule from Digital Lava (http://\vww.digitalIavacom). Some of these such as 
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RealMedia and Video Capsule do not have published technical specifications. Some 
existing formats, such as .exe video files, provide a means for distributing media clips that 
are executable and viewable in a standard operating system without the need for installing 
additional software. In such a file, necessary software components are included in a 

5 single file with a name like movieclip.exe. Generally, such a file is not capable of playing 
streaming data or of varying the components downloaded. 

Downloadable machine code technology (e.g. applets) is also known. Sun 
pioneered aspects of downloadable executable components by creating Java, which has 
become used to make software components downloadable through web browsers to run 

10 on a local virtual machine. Microsoft's COM technology is a later developed technology 
ready to be downloaded from Internet and run in a document container, such as a w^eb 
browser. 

The Internet is a rapidly growing communication network of interconnected 
computers around the world. Together, these millions of connected computers form a vast 

15 repositor\^ of hypermedia information that is readily accessible by users through any of 
the connected computers from anywhere and anytime. As there are an increasing number 
of users who are connected to the Internet and surf for various information, there is 
tremendous demand for more content to be available and methods to deliver content on 
the Internet. Currently, the most commonly available media information that is 

20 deliverable on the Internet may include text information, images and graphics, videos and 
audio clips. 

One known format for this data is streaming data, often used for video and/or 
audio data. For example, streaming video is a sequence of encoded images sent over a 
network or other communications media and presented at a user device, with presentation 

25 of part of the data overlapping some of the receiving of the data. With streaming data, a 
user does not have to wait to completely download a large file before accessing that file 
or parts thereof (e.g. seeing video or hearing sound). As will be understood in the art, 
streaming data, such as video data, can be "live" data, such as streaming encoding of a 
radio or television broadcast, or can be previously stored data, such as pre-recorded audio 

30 or video data. The designation "real-time" is sometimes used in the art to indicate 
streaming data generally, included pre-stored media components. This term is used to 
indicate that delivery of the data is time-critical in that once playback begins, the 
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Streaming data must be delivered quickly enough to keep up with "real-time" presentation 
of the data to the user. Such data is sometimes also referred to in the art as '^clocked 
data" to indicate that the data generally must be delivered at a nearly constant rate. Too- 
fast'delivery^can s\vamp~the' receiving~en'd^s^buffer~capacity. ~ Too "slow deiiveryvviil ~ 
5 cause unacceptable halts or pauses in playback. 

To access streaming data, a user generally uses an access module. This module 
can be either software or hardware, with logic functions able to present the data in a form 
desired by a user. As examples, an access module could be a software player (such as 
RealMedia™) running on a compatible information platform. An access device may also 

10 be more dedicated hardware, such as a cell-phone capable of accessing certain types of 
streaming data, a PDA, a television with the necessary logic, etc. One difficulty with 
standard access methods is that with the increased improvement in video/audio 
compression technologies, access module logic may have to be updated frequently when 
new data formats are used. Further, for media presentations including different types of 

15 data, an access device (such as a general purpose computer or PDA) has sometimes to be 
loaded with several different access modules to access different media. 

SMIL and SMIL Players and Authoring Tools 

To facilitate the creation and distribution of multimedia content on the web, in 

June of 1998, the World Wide Web Consortium (W3C) announced the first specification 
20 of the Synchronized Multimedia Integrated Language (SMIL). This language provides 
some mechanisms for structuring various elements of multimedia content, including 
audio, text, image, and motion video and provides a w^ay for web authors to schedule the 
playing of time-based media within HTML documents. 

A SMIL document according to the specification is actually an ASCII text file, 
25 which authors can edit with almost any text editor. SMIL documents generally resemble 
HTML files and references included components from external URL locations. SMIL 
uses XML to provide the mechanism to define SMIL markup tags for associating timing 
and positioning relationships between multimedia objects, so that audio, videos, images, 
and text can be synchronized. 
30 SMIL enables web authors to position (location, size, z-index ordering) visual 

media objects and assign temporal attributes (begin time, duration or end time) among 
one and another SMIL provides capabilities to create interactive (via hyperlinks) 
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multimedia presentations similar to those on computer-based interactive 'CDs. SMIL 
media objects (addressable by a URL) can either reside locally or be distributed over the 
web. SMIL media objects can be updated independently and remotely. 

A SMIL Player is used to schedule multimedia presentation in a SMIL document, 

5 retrieving media objects on the web using URLs (Universal Resource Locator) described 
in the document, parsing the timing relationship among downloaded media, and 
consequently displaying the presentation based on the correct timeline. The Java SMIL 
Player is a Java applet-based player that follows a platform-neutral programming 
paradigm and makes use of the XML Parser and Java Media Framework (JMF). It can run 

10 within any popular browser without the need to download any plug-ins or ActiveX 
controls. 

A number of SMIL authoring tools have been proposed, which can generally be 
grouped into two categories: general purpose tools designed to create synchronized 
multimedia content for any SMIL-compliant players; and special purpose tools designed 

15 to create synchronized multimedia content for particular players. While these authoring 
tools are powerful, they still require proficiency in SMIL programming, which limits the 
range of users. Currently most SMIL authoring tools in the market have several properties 
in common: they include a text-based editor; they include a WYSIWYG (What You See 
Is What You Get) layout editor; and they do not include a media timeline design wizard. 

20 T.A.G. SMIL Editor 1.0, for example, is a general purpose SMIL authoring tool, which is 
designed to create synchronized multimedia content for any SMIL-compliant players. 
SuperTool is a specific purpose SMIL authoring tool for Real G2 Player only. SuperTool 
has a point and click WYSIWYG layout interface. Developers can add available media 
types, arrange media's layout and sequence how they are played in SMIL composition. 

25 Prior Publications 

The following publications may be related to the invention or provide background 

information. Listing of these references here should not be taken to indicate that any 
formal search has been completed or that any of these references constitute prior art, 

Blakowski, G, and Steinmetz, R. A Media Synchronization Survey: Reference 
30 Model, Specification, and Case Studies, IEEE Journal on Selected Areas in 
Communications, Vol. 14, No. 1, pp. 5-35, Jan. 1996. 
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Synchronized Multimedia Integration Language (SMIL) LO Specification, WWW 
Consonium, Jun. 1998, available at www. w3.org/TRyREC-smil. 

SJLIMMARY _ 

The approach of the present invention is different from aforementioned works in 

5 pan because a Media Missile is open to and can carry different media specification's 
requirements. In one embodiment, not only actions and data embedded in the media 
objects are part of the Media Missile, the Media Missile can also carry executable binary, 
signature, and auxiliary information (which, in specific embodiments, may be pointed to 
by URLs). Unlike the above mentioned prior streaming techniques, which do not 

10 describe carrying various active media content, an executable binary discussed below can 
be carried in a Media Missile to make it ready for a runtime loader on a presentation 
device to run the media missile executable without need for other preinstalled software. 
Thus, in specific embodiments, Media Missile makes uses of operating system's launch 
sequence to run an included executable component and is self-contained for both of the 

15 software components and media data. 

Media Missile, according to specific embodiments of the present invention, can 
benefit from technological advances such as Java or COM, but does not rely on them. 

In specific embodiments, the invention involves a method and^or system for 
providing streaming content that can be activated from a single resource locator and in 

20 response to an activation, begin transferring a streaming presentation in a multimedia 
container, wherein said container has the capability of including one or more executable 
code components, one or more streaming content components, and one or more sets of 
presentation data. A Media Missile launcher in further embodiments may receive from a 
user device data indicating available hardware and software components at said user 

25 device and this data may be used by a scheduler/parser at the user device to request or not 
request different components from the media missile repositor>'. Therefore, a media 
missile can be transmitted with some executable code or no executable code, depending 
on which components are present at the user site. 

In a further embodiment, presenting a media missile according to the invention is 

30 self-directed and requires minimal active management by or presence of preinstalled logic 
components at a user device. 
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In further specific embodiments, the invention involves a method and/or system 
allowing a creator (or author) to specify a streaming presentation including receiving 
indications from an author specifying one or more items of streaming content and 
specifying presentation parameters. The content and parameters may be located at a 
server remote from the author. 

The invention and various specific aspects and embodiments will be better 
understood with reference to the following drawings and detailed descriptions. In 
different figures, similarly numbered items are intended to represent similar functions 
within the scope of the teachings provided herein. In some of the drawings and detailed 
descriptions below, the present invention is described in terms of the important 
independent embodiment of delivering visual and/or audio content. This should not be 
taken to limit the invention, which, using the teachings provided herein, can be applied to 
other streaming data content. For purposes of clarity, this discussion refers to devices, 
methods, and concepts in terms of specific examples. However, the invention and aspects 
thereof may have applications to a variety of types of devices and systems. It is therefore 
intended that the invention not be limited except as provided by the attached claims and 
equivalents. 

Furthermore, it is well known in the art that logic or digital systems and/or 
methods can include a variety of different components and different functions in a 
modular fashion. The following will be apparent to those of skill in the art from the 
teachings provided herein. Different embodiments of the present invention can include 
different combinations of elements and/or functions. Different embodiments of the 
present invention can include actions or steps performed in a different order than 
described in any specific example herein. Different embodiments of the present invention 
can include groupings of parts or components into larger parts or components different 
than described in any specific example herein. For purposes of clarity, the invention is 
described in terms of systems that include many different innovative components and 
innovative combinations of innovative components and known components. No 
inference should be taken to limit the invention to combinations containing all of the 
innovative components listed in any illustrative embodiment in this specification. The 
functional aspects of the invention, as will be understood from the teachings herein, may 
be implemented or accomplished using any appropriate implementation environment or 
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programming language, such as C+^, Cobol, Pascal, Java, Java-script, etc. All 
publications, patents, and patent applications cited herein are hereby incorporated by 
reference in their entirety for all purposes. 

The invention therefore in specific aspects provides a streaming video/audio 
5 signal that can be played on various types of video-capable terminal devices operating 
under any types of operating systems regardless what type of players are preinstalled in 
the temiinal devices 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram showing an overall architecture of a system showing 

10 both authoring and playback according to various embodiments of the invention including 
multiple novel elements. 

FIG. 2 illustrates an exemplary configuration of a network in which the present 
invention may be practiced. 

FIG. 3 illustrates an example media missile container format according to specific 
15 embodiments of the invention. 

FIG. 4 is a flow chart illustrating functional aspects of a multimedia player 
according to a specific embodiment of the invention. 

FIG. 5 is a flow chart illustrating an example method for accessing a media 
missile according to a specific embodiment of the invention. 
20 FIG. 6 illustrates the operation of different timers initiated by a scheduler 

according to specific embodiments of the invention. 

FIG. 7 shows a standard browser display of example source files that may be 
packaged into a media missile in an example in accordance with specific embodiments of 
the present invention. 

25 FIG. 8 shows an example interface for packaging source data into a media missile 

according to specific embodiments of the present invention. 

FIG. 9 illustrates an example general metadata only media missile format 
according to specific embodiments of the invention. 

FIG. 10 illustrates an example metadata only media missile data structure 
30 according to specific embodiments of the invention. 
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FIG. 1 1 illustrates an example media missile data structure according to specific 
embodiments of the invention. 

FIG. 12 illustrates an example binary encoded metadata only media missile data 
according to specific embodiments of the invention. 

FIG. 13 illustrates an example presentation of a media missile in a player 
according to specific embodiments of the present invention. 

FIG. 14 illustrates an example general method for guiding a user in creating a 
media missile or presentation data according to specific embodiments of the invention. 

FIG. 15 illustrates an example of a Synchronized Authoring Manager (SAM), also 
referred to as a Synchronized Authoring Tool (SAT), according to specific embodiments 
of the invention. 

FIG. 16 illustrates an example of a Preview Manager GUI according to specific 
embodiments of the present invention. 

FIG. 17A-C illustrate examples of choosing various presentation templates 
according to specific embodiments of the present invention. 

FIG. 18A-B illustrate adding and previewing media according to a Video Lecture 
Template according to specific embodiments of the present invention. 

FIG. 19A-B illustrate adding and previewing media according to a Headline News 
Template according to specific embodiments of the present invention. 

FIG. 20A-B illustrate examples of creating presentation specification metadata 
according to specific embodiments of the present invention. 

FIG. 21 illustrates an example of a Document Generation GUI according to 
specific embodiments of the present invention. 

FIG. 22 is a block diagram showing a representative example logic device in 
which aspects of the present invention may be embodied. 

DESCRIPTION OF SPECIFIC EMBODIMENTS I 

In specific embodiments, the present invention involves a streaming media 

structure suitable for providing multimedia streaming over a communication data network 
including a cable network, a local area network, a network of other private net\vorks and 
the Internet. The detailed description of the present invention include numerous specific 
details that are set forth in order to provide a thorough understanding of the present 
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invention. However, it will become obvious to those skilled in the art that the present 
invention may be practiced without these specific details. In other instances, well known 
methods, procedures, components, and circuitry have not been described in detail to avoid 
"uhnecessTriiy obscunng'aspecty " ~ 

5 The present invention is presented largely in terms of procedures, steps, logic 

blocks, processing, and other symbolic representations that resemble data processing 
devices. These process descriptions and representations are the means used by those 
experienced or skilled in the art to most effectively convey the substance of their work to 
others skilled in the art. The method along v/ith the system to be described in detail below 

10 is a self-consistent sequence of processes or steps leading to a desired result. These steps 
or processes are those requiring physical manipulations of physical quantities. Usually, 
though not necessarily, these quantities may take the form of electrical signals capable of 
being stored, transferred, combined, compared, displayed and otherwise manipulated in a 
computer system or electronic computing devices. It proves convenient at times, 

15 principally for reasons of common usage, to refer to these signals as bits, values, 
elements, symbols, operations, messages, terms, numbers, or the like. It should be borne 
in mind that all of these similar terms are to be associated with the appropriate physical 
quantities and are merely convenient labels applied to these quantities. 

According to one aspect of the present invention, a streaming media structure is 

20 described below. The streaming media structure is provided to create, deliver, 
reassemble, and render synchronized multimedia content with its lightweight machine- 
aware components through streaming services. Each stream represents a user access to a 
video/audio title provided in a host from a terminal device, 

FIG. 2 illustrates an exemplary computer communication configuration in which 

25 the present invention may be practiced. Central server 102 together with a video/audio 
database 104 is a video/audio source comprising video/audio files that can be accessed on 
demand. As used herein, video/audio files or titles are referred to any video footage, 
video films and/or video/audio clips that can be compressed or packed in any format. One 
of the exemplary formats is Moving Picture Experts Group (MPEG). To play MPEG 

30 video files, one needs need an MPEG viewer or a client software executing in a personal 
computer with sufficient processor speed, intemal memory-. Another one of the popular 
formats for audio is MPEG-1 Audio Layer-3 (MP3), a standard technology and format for 
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compressing a sound sequence into a very small file (about one-twelfth the size of the 
original file) while preserving the original level of sound quality when it is played 
through a client program (player) downloadable from many web sites. It should be noted, 
however, that the exact formats of the video files do not affect the operations of the 

5 present invention. As will be noted and appreciated, the present invention applies to any 
formats of the video or audio files as well as other multimedia data that may include, but 
not be limited to, binary data and files, hypertext files and scripts. 

Preferably, data network 106 is a data network backbone, namely a larger 
transmission line. At the local level, a backbone is a line or set of lines that local area 

10 networks connect to for a wide area network connection or within a local area network to 
span distances efficiently (for example, between buildings). On the Internet or other wide 
area network, a backbone is a set of paths that local or regional networks connect to for 
long-distance interconnection. Coupled to data network A 106, there are two 
representative proxy servers 108 and 110 that each service representative terminal 

15 devices 116-119 via data network 1 12 and 1 14 respectively. It should be noted server 102 
sometimes named "central" or "central video" server does not necessary mean that server 
102 is only ser\^ed as a repository of all the video titles. In some case, server 102 may 
ser\ice terminal devices directly and may retrieve other video/audio files from other 
ser\^ers such as proxy servers 108 and 1 10. 

20 Data netvv'ork 112 and 114 are typically the Intemet, a local area network or 

phone/cable network through which terminal devices can receive video files. The 
terminal devices may include, but not be limited to, multimedia computers (e.g. I 16 and 
1 19), networked television sets or other video/audio players (e.g. 1 17 and 1 18). Typically 
the terminal devices are equipped with applications or capabilities to execute and display 

25 received video files. For example, one of the popular applications is an MPEG player 
provided in WINDOWS 98 from Microsoft. When an MPEG video file is received in 
streaming from one of the proxy ser\^ers, by executing the MPEG player in a multimedia 
computer, the video file can be displayed on a display screen of the computer. 

According to one aspect of the present invention, the streaming media structure is 

30 provided to create, deliver, reassemble, and render synchronized multimedia content with 
its lightweight machine-aware components through streaming services. 
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Cases 1 and 2 below present detailed procedures of how to construct the streaming 
media structure relying on a complied and linked version implementing the present 
invention and executing preferably in a client device. As such a video/audio title can be 
submittedTo~arh'ost"7ievic^ a^ Torroat' compliant to^ the streaniing" media strilctiireT Xases~ 
5 3 and 4 below present detailed procedures of how to playback video/audio titles packed in 
a streaming media data structure according to embodiments of the present invention. 

It can be appreciated by those skilled in the art that one of the important features 
in the present invention is a new versatile streaming data file format or structure that can 
be accessed from any types of terminal devices operating under any types of operation 

10 systems. Another important feamre is that the present invention, in specific embodiments, 
allows the elimination of preinstalled players. A streaming media structure according to 
specific aspects of the invention can include a built-in player that is downloaded on the 
fly and then operates in a local machine to access the streaming data and to find and 
install any necessary additional executable components. As will be understood to those of 

15 skill in the art from the teachings herein, elimination of preinstalled players for various 
types of streaming content allows streaming content to be more easily accessed on 
devices with limited code storage, such as cell-phones or other hand-held or versatile 
playback devices. 

The processes, sequences or steps and features discussed herein are related to each 
20 other and each are believed independently novel in the art. The disclosed processes and 
sequences may be performed alone or in any combination to provide a novel and 
nonobvious file structure system suitable for media delivery system. It should be 
understood that the processes and sequences in combination yield an equally 
independently novel combination as well, even if combined in their broadest sense. 
25 The streaming media structure as described herein, in accordance with one aspect 

of the present invention is robust, operationally efficient and cost-effective. The 
mechanism provides the best use of servers providing streaming ser\'ices and permits 
seamless delivery of streaming video with highest quality of services possible. In 
addition, the present invention may be used in connection with presentations of any type, 
30 including sales presentations and product/service promotion, which provides the video 
service providers additional revenue resources. 
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In order to facilitate description, the following discussion will describe the present 
invention in terms of streaming media presentations. It will be understood to those of skill 
in the art, however, that the invention may be used in other streaming data situations. The 
invention should therefore not be taken as limited except as provided in the attached 
claims. 

In various embodiments, systems or methods according to various aspects of the 
present invention, can create, deliver, reassemble, and render synchronized streaming 
multimedia content with lightweight machine-aware components through streaming 
services. Each stream represents a user access from a client on one computer to a server 
on a different computer. As will be understood from the teachings herein, a system or 
method according to the present invention can handle various types of streaming data 
including real-time multimedia content as part of a presentation or multimedia missile 
through streaming services. 

In particular embodiments, an author (a user or a program) creates synchronized 
presentation using a wizard-like Synchronized Authoring Manager (SAM). . -In further 
embodiments, an SAM can include lightweight software components and/or indications 
of such components with the media data content into a Media Missile container that is 
streamable from a single location. 

The present invention, in some embodiments, thus allows that the streaming of 
active and passive content can be done all at once and can be self-contained and 
guidance- free. The active content is the various machine-aware runtime components. The 
passive content is the media data to be rendered at runtime. All of the presentation 
content (both active and passive) can be stored in a container-like file format or object- 
relational database, A special header can be attached to individual media objects to 
specify metadata information such as signature, data length (or offset), and linkage. A 
Presentation Specification can be generated in descriptive tagged language such as 
Synchronized Multimedia Integration Language (SMIL) from World-Wide Web 
Consortium (W3C). 

A container-like file format of a Media Missile has advantages for content 
providers in the convenience and portability of the content. With the flexible file 
specification of the Media Missile, a content provider of a synchronized Multimedia 
presentation can create the presentation and then store, manage, distribute, or serve the 
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presentation as easily as they would a single file. Thus, a synchronized muliimedia 
presentation, using a media missile, can be as easy to deliver or manage as a standard GIF 
or JPEG banner ad. Conventional SMIL-based or HTML-based presentations cannot 
merge mLnrrpl^objectTinto a singre~conFai'ner and"can require managing alKillownlbaaing" 
5 dozens of individual files to render properly, 

A media player implementation, such as a an SMIL player, that is compliant to 
this design can render and playback media contents in real-time in accordance with a 
presentation specification. In particular, individual media content objects, specified in the 
presentation, corresponds to object information in the metadata and media data in the 
10 media missile. 

In a further embodiment, a one-way one-to-one hashing function or other method 
can be used to generate a signature for the media. The signature can be both a watermark 
of the media, and also be a search key (to be used in place of a name) if the media is 
stored in a large distributed media repository'. In a further important embodiment, a media 
15 missile format enables synchronized presentation on mobile platform with limited pre- 
installed software capability or memory' footprint. 

FIG. I is a block diagram showing an overall architecture of a system showing 
both authoring and playback according to various embodiments of the invention including 
multiple novel elements. Various aspects and details of the designs are elaborated in later 
20 sections. The overall architecaire shown in FIG. I includes the following functional 
and/or structural elements. 

Media Missile (MM) 12 is a container specification that allows inclusion of 
various items of machine-aware executable binary (or script), media data, and associated 
information such as presentation specification or metadata. (MM) can be a container of 
25 container(s) in a nested structure. The machine-aware executable binary (or script) is can 
also be referred to as active media content. The active media content can be implemented 
in machine-neutral programming paradigm (such as Java) or machine-dependent code 
(such as .exe files). Media Missile with active media content is self-contained and 
guidance-free for synchronized real-time multimedia authoring and/or presentation over 
30 netw^ork environment. Once the media missile is indicated and the active media content is 
retrieved and loaded to a machine, it is ready to run by itself 
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Synchronized Authoring Manager (SAM) 10 takes and understands various 
language specifications, formats, and templates in synchronized multimedia, including 
Media Missile. Language specification and templates such as SMIL-compliant ones are 
used as input to SAM's authoring wizard, hi specific embodiments, an SAM wizard 
5 guides a user to compose a synchronized presentation and retrieve media data from Media 
Repository Manager 20. hi one option, an end result of an SAM is a packaged Media 
Missile. As an option, an SAM 10 can also output a separate presentation specification 
(such as an SMIL presentation specification) that is compliant to international or industry 
standards. 

10 As shown in FIG. I, SAM 10 can receive as inputs templates, media missile 

specification, multimedia language specification, and media data, which may be received 
from a media repositor\\ Templates, as understood in the art, include data describing a 
general format, layout, or structure for a presentation. SAM 10 creates a media missile 
and/or presentation specification, which is stored in repository 20. 

15 Media Repository Manager (MRM) 20 handles storage and retrieval of real- 

time (streaming) and non-real-time data accessible by a client by activating (or 
*'shooting'0 a media missile. The real-time data (or clocked media data) 22, in a specific 
example, can refer to media data or media missiles stored or broadcast from Streaming2rs 
Media Server and Caster or other compatible video server platforms, which are 

20 collectively illustrated as data source 30. The non-real-time data is the presentation 
specification or any other auxiliary information stored on a web server or file serv^er. In 
the figure, Data Store 40 illustrates generally one or more stores for additional digital 
data. 

As shown in FIG. 1, MRA4 20 can output either directly to an SMP 50 or through 
25 a web or other front-end interface 55. Also, as shown, MRM 20 acts as a middle manager 
for communicating media data with SAM 10 and managing data from source 30 and data 
store 40. 

(MRM) 20 also can be understood as representing access to any media data that is 
available to SAM 10 and SMP 50, such as any authorized and available computer on the 
30 Internet, 

Synchronized Multimedia Player (SMP) 50 has an internal scheduler to process 
delay-sensitive data. It can also handle sequencing of static media data such as images 
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and text. It complies with the SMIL recommended by \V3C. Hence, it is able to interact 
with web servers to retrieve and/or to utilize SMIL presentation specifications. In 
addition, an SMP can also work with a Media Missile locally on a single platform or in a 

ci i s t"r i bu te dlie t work erw iron n^^^ - — 

As shown in FIG. I, SMP 50 can receive data directly from MRN'I 30, or through 
interface 55. An SMP 50 can also be directed to receive media data directly from a data 
store 40. 

Media Missile 

FIG. 3 illustrates an example media missile container format according to specific 
embodiments of the invention. This example shows a very specific example bit-wise 
arrangement of various elements. It will be understood to those of skill in the art from the 
teachings herein that variations of the field and bit arrangements and formats exist within 
the scope of the invention. Particularly, logic tme may be encoded as a variety of 
different bit values and field lengths may vary, included using more than one bit for fields 
indicated as bit fields below. The container format shown in FIG. 3 is further described 
below. 

Media Missile 12 is a document container, which can include active content and 
passive media content. The active content is machine-aware executable binary or script. 
Passive content can include a presentation specification, metadata information, and the 
media data. The media data can be another Media Missile. Thus, Media Missile has a 
nested structural format. All of the content of a media missile can be stored in a file, 
directory structure, or database, such as on a serv er system or computer such as repository 
303. In a further embodiment, according to specific implementations, some or all 
components can be generated on the fly by a multimedia missile delivery system. 

The executable binary (or script) 200 and the presentation specification 210 are 
both optional. If machine-executable code is not present, E-Length 205 is zero. The 
Continuation Bit 202 tells what to expect for the follow-on Media Missile structure. If 
Continuation Bit 202 is one and E-Length 205 is zero, this indicates that the entire Media 
Missile starts from a presentation specification (210). If Continuation Bit 202 is zero and 
E-Length 205 is zero, it indicates that the entire Media Missile starts from MetaData 
information 220. Finally, if Continuation Bit 202 is zero and E-Length is "-1", the entire 
Media Missile consists of pure media data. 
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Continuation Bit 212 in front of P-Length 215 tells whether the metadata 
information is included in the media missile (1) or not (0). If no metadata information is 
available, the data right after Presentation Specification is the media data. 

If metadata information 220 is present, the first bit 222 tells whether the data right 
5 after the metadata information is another Media Missile (1) or just pure media data (0). 
in metadata infonnation, the signature field 226 is 128-bits by default to store a digital 
signature using one-way one-to-one hashing function such as MD4 (Message Digest ,4) or 
MD5 (Message Digest 5). The length of the signanare field can also be implicit by the 
implementation to guard against trivial breakdown and guessing. 

10 Following signature field 226, MetaData 220 holds one or more length field(s) 

(such as 245 and 295) for media data. Each length field is 64-bit starting with a bit 
representing URL (1) or payload (0) (such as 242 and 292). In specific embodiments, 
each length field in MetaData 220 corresponds to a segment of data in media data 230. 
The actual data stored in 230 can be characters indicating a URL, or can be the media 

15 data. Segment(s) of data are stored in sequence according to the sequence of length 
field(s). Here, URL stands for Universal Resource Locator for World-Wide Web 
hyperlink; payload indicates media data for the media player. As will be understood in 
the art, however, URL generally can refer to a locator (such as a filename or database id) 
for any resource available via a computer or communications system. In specific 

20 embodiments, a URL or other locator can indicating streaming or ''real-time'' data 
sources, such as sources of live streaming data. 

As will be understood from this discussion and other descriptions herein, an 
important and innovative aspect of the Media Missile Container Specification is that once 
a user '"shoots" the Media Missile from a particular location, from the perspective of the 

25 user, the contents of the Media Missile can be delivered apparently from a single location. 
In contrast, with formats such as a standard HTML or SMIL type page, a client at a user 
site may have to access dozens of different text, static image, executable, or audio and 
video files, from dozens of different locations, to complete a presentation. With a Media 
Missile, a client can access all of the included data from a single location, with the data 

30 delivered in the Media Missile format. This aspect also has advantages for data providers 
in that Media Missile formatted data is easier to manage, distribute, and deliver than is a 
presentation the requires many different files. 
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Examples of Authoring Media Missile 

Wliat follows are descriptions of the operation of authoring manager 10 according 

to specific embodiments of the present invention. A wizard, as discussed below, is 

understood— in-the-ai=t— as— logic— implemented-proeesses-that—guides™a— user— through^^ — 

5 sequence of steps in an assisted and constrained fashion to accom.plish a result, such as 
installing software or creating a particular document or file type. A template, as discussed 
below, is understood in the art as set of data specifying general format parameters that 
may be used to determine or guide the final format of one or more final files or data 
structures. Synchronization primitives are further described below and illustrated in FIG. 
10 6. 

Example Basic Authoring Method 

The steps to create or update a Media Missile (using Synchronized Authoring 

Manager (SAM) 10 are as follows. 

1. The Synchronized Authoring Manager (SAM) provides template selection for 
15 user to create a synchronized multimedia presentation. It can also read from an existing 

presentation specification for user to update the specification. 

2. SAM takes presentation specification and media data as input used to parse and 
create a tree representation internally. The tree is a top-down representation of the 
synchronization primitives. Each intermediate node is either a parallel or sequential 

20 synchronization primitives. SAM then retrieves or calculates the required fields according 
to the Media Missile specification for the leaf nodes (individual media) in the tree. This 
can be a pipeline process. 

3. SAM attaches system-required fields, e.g., a 64-bit trailer, to individual media 
specified in the presentation to become a basic Media Missile with media data only. 

25 4. SAM uses a depth-first tree traversal to calculate and create optional fields 

based on user input. The optional fields are feasible to apply to any node in the tree 
depending on user's needs. 

5. SAM uses a breadth-first tree traversal to traverse the entire tree and create the 
final Media Missile. 
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Sample Sequences of Authoring Interactions 

Case 1: Creating Synchronized Multimedia Presentation for a Guided Lecture: 

1 . A wizard helps author to pick up a lectxire template or other template of the 

presentation skeleton through a point-and-click list, where corresponding layout of the 
template also displays. 

2. The wizard asks the author to pick up a lecture video and corresponding 
presentation slides in sequence. 

3. The author has a choice to use default duration for slide display duration or use 
MetaData Manager to customize the presentation timeline. 

4. If using the MetaData Manager, the wizard displays both the video and 
con-esponding slides at a particular point in time. Author has the liberty to modify the 
duration setting for each slide, 

5. The wizard then prepares the final output, both the presentation specification 
and media object, in user-specified formats, e.g., SMIL for presentation specification or 
Media Missile for media object. 

6. The wizard asks author whether author wishes to keep the final output locally 
or upload it to a specific hosting server. 

Case 2: Updating Synchronized Multimedia Presentation for a Guided Lecture: 

1. Author uses wizard to pick up an existing presentation specification or media 

missile from a hosting server, 

2. The wizard asks the author to delete or add video and slides. 

3. The author has a choice to use default duration for slide display duration or use 
MetaData Manager to customize the presentation timeline similar to step 3 in Case 1. 

4. The wizard repeats step 4 and 5 in Case 1 . 

5. The wizard asks the author to apply the changes and to check if object deleted 
from the specification should also be permanently removed from the hosting server(s). 

6. The wizard repeats step 6 in Case 1 . 

Example Basic Method for Playing Back Media Missile 

Playback of Media Missile according to specific embodiments of the present 

invention involves three potentially overlapping processes: delivering, assembling, and 
rendering Media Missiles and their contents. 
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Example Method for delivering a Media Missile are as follows: 

1. User can use a browser to point-and-shoot the Media Missile. Alternatively, 
user can also use command line interface to launch a local self-contained Media Missile, 

or run a pla yer with a Media Missile name as input. 

5 2. If the' Media Missile contains a complete executable binary (or script), it 

generally should be able to load itself and run (e.g. it can be loaded by the native 
operating system of the target device). In specific embodiments, Media Missile takes 
advantage of the runtime loader characteristics available on most operating system 
platforms such as UNIX or Windows. For example, runtime executables or scripts are 

10 identified by a Magic Number (first two bytes in any file) on UNIX platforms. On 
Windows platforms, runtime executables and scripts are explicitly identified by a three- 
letter filename extension registered in the system registry (such as .EXE or .COM). A 
loader generally also checks to determine if the files are indeed executable. Other 
operating system platforms may have different mechanisms to load and launch runtime 

15 executables that may be used by a media missile. 

3. The Media Missile will be streamed and downloaded from an application server 
or a web server if it is at remote site. A browser for a web or application server is 
generally able to run or save a remote file. To make Media Missile compatible with these 
servers, a hyperlink retrieval of a Media Missile in a web-enabled page will implicitly 

20 stream only the machine-aware portion of the Media Missile. The machine-aware portion 
can then be launched and run on a local platform and manage requests for additional 
portions of the media missile. Because the Media Missile's executable footprint size is 
known ahead of time, at runtime the executable is able to manipulate the rest of the Media 
Missile according to its specification. 

25 Example Method for presenting a Media Missile are as follows: 

1. The Synchronized Multimedia Player (SMP) can be locally-resident executable 

code or executable code that is downloaded or streamed with the Media Missile. SMP has 
a timeline-based scheduler to schedule the playback according to the presentation 
specification. The presentation specification again may be available within the Media 
30 Missile or from another source, such as through web page that is holding the Media 
Missile. 

2. SMP parses the presentation specification to build a presentation tree. 
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3. SMP fires the presentation from the top of the tree based on the synchronization 
characteristics (cither parallel or sequential) of each intermediate tree node. For parallel 
nodes, SMP creates independent execution threads to handle each parallel task and each 
of these threads refers to a global time base. For sequential nodes, SMP can use a single 
execution thread to step through the sequential tasks. 

4. When SMP fires a leaf node, SMP locates and launches the media player and 
media data. 

If the link to the media data in the specification is a relative path, SMP will by 
default search the Media Missile itself, using the media data's signature or other identifier 
to see if the media data is available in the Media Missile. This searching can take place on 
a Media Missile either that exists already at a local machine or on a Media Missile still 
resident at a remote server. Various protocols are known for a streaming data client to 
make requests and receive responses from a streaming ser\^er and the protocols can be 
used to search for specific data within the Media Missile. Otherwise, SMP will follow an 
absolute path either specified in the Media Missile or from the root of the serx^er where 
the Media Missile or presentation specification was streamed, to locate the media data. 

Regarding the player, SMP will look to determine if the player for the particular 
media data is located on its local machine. If the player is not located on its local 
machine, SMP wall search for the player in the Media Missile or in other data store. 

Playing a Media iVlissile 

FIG. 4 illustrates generally the functional elements the enable MM playback 

according to specific embodiments of the invention. As shown in the figure, MM 
playback may access media and code data from a variety of locations. A media missile 
repository 303 can be understood as a primary data storage of media missile data, 
including passive data and active code data. The repository in specific implementations 
can be understood to reside on a server, accessible via a local netw^ork, wide area 
network, or the Internet. When residing on a server (as opposed to locally stored), 
repository can be understood to have capabilities of streaming session servers, including 
seek to end, search, play and pause capabilities in response to requests from a client. In 
some embodiments, the repository will generally store active and passive content needed 
to play a media missile on a variety of platforms, though only content needed for a 
particular platform may be downloaded from 303 by an SMP, 
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Local Code Resource 305 can be understood as those code components available 
at the local device. For some devices, such as a computer workstation on a LAN, the 
local code resource 305 may be quite extensive. For such devices, a MM scheduler and a 
>7aneTy~df~players^m^y^ " 

5 and therefore not have to be transferred from a serx^er to play a media missile. For other 
devices, such as a cellular telephone for example, there might be very little or no local 
code resource and all code will need to be downloaded from 303. 

Web or Application Server Repository 307 can be understood as storage for any 
active or passive data not in 303 or 305. As described herein, this data may include data 

10 referenced by a resource locator (such as a URL) within a Media Missile. An example of 
such data would be an HTML page or data used to complete a presentation, advertising 
banners or data from external sources, or any other data referenced in the medial missile 
or needed to complete playback. 

As elsewhere described herein, to begin playing a media missile, an executable 

15 binary or script component able to parse a MM is first initiated on a playback device. 
This executable, as described herein, may be initially loaded from 303, 305 or 307. An 
initial task, in particular instances, is to parse a presentation specification 402, if present, 
and then to locate and execute a Real-Time Presentation Scheduler 404. Scheduler 404 
then schedules the dov/nloading (if necessary) and initializing of any players 406 needed 

20 to playback the actual media data. These players can include a variety of off-the-shelf 
media players, such as QuickTime'^^^ or RealMedia'^^* players or the players can include 
proprietary players. As indicated in the figure, once scheduler 404 has initialized, it 
schedules the downloading and initialization of any players. Thus, the downloading and 
initializing of various player and player components is under control of the overall 

25 presentation schedule being run by the scheduler. 

As shown in the figure, scheduler 404 also determines the ending of a presentation 
and terminates the necessary components. 

The steps to render or playback a Media Missile are as follows: 

1. Software (SW) checks if the Media Missile contains an initial executable code 

30 portion (binary or script) for the platform. If the checking is positive, go to Step 2. If the 
checking is negative, goto Step 3. This checking can be accomplished by software 
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registered at the target device to handle a MM file type or the checking can be self 
starting if the MM itself is an executable file or script. 

2. Since there are machine-aware components in the Media Missile, it can be 
launched on the fly, or it may be used as an enhancement of the SW already present of the 
system to render the media data within the Missile. 

3. In case there's no machine-aware executable within the Media Missile, the 
media missile or Operating System will trigger and launch a local SMP with default 
components to read media data from the Media Missile. 

4. SMP handles both barrier and event synchronization with elastic timers for 
synchronization point. 

SMP internally has to handle synchronization events according to the presentation 
specification 320. Certain specifications define precise timing sequence. Some others 
have relaxed schedule. Any playback of media(s) can be either sequential or concurrent 
(parallel). 

Sample Sequences of Playback Rendering 

Case 3: Playing Back a Synchronized Multimedia Presentation for objects in Media 
Missile format in a Web Page: 

1. Client (SMP) 50 accesses a synchronized multimedia presentation, e.g., 
formatted in a HTML/XML web page. 

2. The names/signatures of the media objects at particular points in time in the 
presentation timeline are passed to the player(s) in the web page through a player 
scheduler which is downloaded on the fly or resides on the local computing device. 

3. Each player starts to retrieve a media missile header corresponding to the media 
object name/signature. 

4. Each player inspects the header to verify the necessity of downloading and 
registering embedded lightweight software components in the media missile to render the 
media data. 

5. Each player realizes and loads media-specific rendering components at runtime. 

6. For a parallel start of rendering, the scheduler sets a barrier synchronization 
primitive, further description of which is provided below. 

7. Each player is enabled to play back and render media data according to the 
"Enable" signal from the player scheduler, the starting time, and the duration in the 
presentation specification. 
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Case 4: Flaying Back a Synchronized Multimedia Presentation for objects in Media 
Missile format in a distributed environment: 

1. A self-contained guidance-free Media Missile, which is downloaded or stored 
in local storage, is launched. The beginning of the Media Missile is the player scheduler, 

5 which can be a script or a binary executable object compatible with the runtime platform. 
Hence, it would run by itself. 

2. The player scheduler then loads a presentation specification, which is attached 
after any present runtime executable code segment in the Media Missile. 

3. The names/signatures of the media objects and their position in the Media 
10 Missile at a particular point in time in the presentation timeline are passed to the player(s) 

specified in the Media Missile through the player scheduler. The player(s) (which can 
also be a Media Missile by itself) resides in a distributed network environment. Hence, 
the name of a media object can be a link to a Media Missile over the networks, 

4. The procedure then repeats step 3 to 7 in Case 3. 

15 FIG. 5 is a flow chart illustrating an example method for accessing a media 

missile according to a specific embodiment of the invention. 

Playback Timing 

To handle both of the sequential or parallel playback of media contents with either 
precise or elastic timers, an SMP defines the following states: started, initialized, played, 

20 paused, and stopped for each media player under its control. FIG. 6 illustrates the use of 
these states with the synchronization primitives and the timing requirements. SMP 
attached a timer for each synchronization primitive to control the duration. Each media 
player may also have attached timers to control both delay and duration. Whether the 
content has additional embedded timing information or not, it can be played back as if it 

25 is a continuous media, i.e., media with timing information. Thus, SMP turns some static 
media types into time-based media such as text or image. The implementation of the 
synchronization checkpoints in the Real-Time Presentation Scheduler at any point in time 
is elaborated as follows and as illustrated in FIG, 6. 

Bl. Sequential Playback with Precise Timer: 

30 In this case, the scheduler 330 pairs the started and the stopped time of any two 

media players 340. As such, a player may be launched, based on estimated latency, to 
cache (or buffer) media data before it starts to render the data. 
B2, Parallel Playback with Precise Timer: 
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In this case, the scheduler checks for barrier synchronization on started, 
initialized, played, and stopped states for all of the players. Each player may be toggled 
between played and paused states if delay timers are set within the player, 

B3, Sequential Playback with Elastic Timer: 
5 In this case, the scheduler pairs the started and the stopped time of any two media 

players. However, one must be stopped before another is started. Altematively, it can be a 
single media player that takes sequential input scheduled in time. 

B4, Parallel Playback with Elastic Timer: 

In this case, the scheduler only sets barrier synchronization for started and stopped 

10 states for all of the players. Each player controls its own status and the scheduler ignores 
any events from players other than started and stopped events. 

As will be understood to those of skill in the art from the teachings herein, the 
synchronization of multiple players allows the presentation of multi-part media 
presentation. Case B2, for example, might illustrate a case where a presenter or lecturer 

15 is narrating a video clip or a slide-show presentation. The presenter will come to a point 
in the presentation where it is time to begin the video clip, the clip will begin, and the 
presenter audio or audio and video, will continue while the video clip proceeds. The 
video clip might occupy an entire screen space, or the clip might occupy a subwindow in 
the screenspace, while video of the presenter occupies a different window portion. As 

20 shown in FIG. 6, initialization of a Player2 can begin at some point during the activation 
of Player L Thus, Player2 or Player3 components need not be downloaded early and 
might not be downloaded at all if a user halts ends access to a presentation. It will be 
understood from the teachings herein, as discussed with respect to FIG. 3, that the format 
of the media missile allows a multi-player presentation to continue from a media missile 

25 location, downloading lightweight software components as needed according to different 
timer strategies as shown in FIG. 4. 

Example Application (WebAds) 

Further understanding of aspects of the present invention will be provided by 

considering the following example. This specific example is provided to illustrate 
30 various aspects of the present invention, but the invention should not be limited by any of 
the details provided herein. This example involves use of the herein described media 
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missile format, according to specific embodiments of the present invention, to enable a 
WebAds streaming presentation. 

Using A Packaging Utility to Packatie a MM 

As"a~simpie-exampie— a"MM""can-be created-fronran"existing"MTMt presentation 

as described below. In a particular embodiment, a package utility may be launched as 
follows: 

package webAdsUrl [webAdsDir startingHtmFile] 
hi this example, webAdsUrl indicates a URL pointing to an external HTML 
location, in case the remainder of the packaged file is corrupted or emptv^ when the MM is 
accessed. webAdsDir indicates an absolute or relative path to specify a directory the 
includes files to be placed in the MM and startingHtmFile indicates an initial html file 
that will be used as the initial outline of an MM. As an example, consider a Web Ads 
directory that includes the following files: 

c:\webAds\s2 1 WebAds. htm 

c:\webAds\otherLinks.htm 

c:\webAds\logo.gif 

c:\webAds\sponsor\abcCo.jpg 
In this example, the package utility can be run with a command line such as: 

c: \s21 \bin\package "http://w\v\v.abcCo.coni " cUvebAds s2 IWebAds.htm 
This example can be further understood with reference to FIG. 7 to FIG. 13. FIG. 
7 shows a standard browser display of example source files that may be packaged into a 
media missile in an example in accordance with specific embodiments of the present 
invention. According to various specific embodiments of the present invention, and as 
will be understood from the teachings provided herein, this source data file could be any 
type of media data. Here, the example media data includes an HTML file and its 
included image files, including a GIF and a JPG file. The HTML file also includes a 
reference to another HTML file, otherLinks.htm, which could include any additional links 
or hlml-type data. 

FIG. 8 shows an example interface for packaging source data into a media missile 
according to specific embodiments of the present invention. A variety of different 
packaging tools can be used in different embodiments of the invention, including 
advanced wizards for creating MM as described herein, in this example, however, a 
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simple packaging tool is used to create VVebAds, and this packaging tool requests just the 
four parameters illustrated. 

When playing such a file, a player can extract the packaged file and place it into a 
temp directory. If the file is an audio-only clip, a default HTML browser page can be 
5 displayed, or the staning html file (e.g. s21WebAds.htm, as shown in FIG. 13) can be 
displayed in the player window. When the player quits, it generally will remove the 
temporary files and temporary directory. 

If player.dat is missing, a player can use a centrally located file (such as 
vvww.Streaming21.com/ads) or show error and then stop the application. If player.dat is 
10 corrupted or player.dat contains no files, a player will use the default URL inside the 
player.dat, if possible. In the worst case, the player can display the default bitmap for 
Audio playback. 

FIG. 9 illustrates an example general metadata only media missile format 
according to specific embodiments of the invention. In this example, the file begins with 
15 metadata information. As indicated in the figure, the metadata includes one or more 
URLs for associated files with the streaming presentation. Following the metaData is the 
mediaDataBlock, which contains the actual media object data or locators for media object 
data. 

FIG. 10 illustrates an example metadata only media missile data structure 
20 according to specific embodiments of the invention. Note that in this example, the values 
shown at the top of the figure are assumed. Note that as indicated in the figure," all of the 
source files are incorporated into the media data area of the media missile file, with the 
metadata area indicating the identity of the portions of data in the media data area. 

FIG. 1 1 illustrates an example media missile data structure according to specific 
25 embodiments of the invention. Comparing this figure with FIG. 3 will show one specific 
data structure example of how a Media Missile according to specific embodiments of the 
present invention may be realized. Note that objects in 

FIG. 12 illustrates an example binary encoded metadata only media missile data 
according to specific embodiments of the invention. 
30 Optional processing for WebAd in a media missile allows ignoring an Executable 

Binar>' of Script portion and a Presentation Specification portion, if either are found in the 
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media missile as, for example, shown in FIG. 11.. An example of seeking in a media 

missile file to skip this data is as follows: 
OpenFi le 

Seek to endin g- Sb yces 

"BOOL continat ion_bi t = readlBit:(); 

int64 eLength = readNextBi ts ( 6 3 ) ; 

SeekBackToBeginning ; 
if (eLength==-l) 

error // required metaDataBlock is absent 

else if (eLength) 

skipFirstBytes ( eLength) ; // skip Executable Block 
if (p_blk_exist) 

{ BOOL next_blk_is__meta_data = readlBit() ; 
if ( ! next_blk_is_meta_data) error; // required metaDataBlock is absent 
else 

{ int64 p_blk_len = readNextBi ts ( 6 3 ) ; 

skioFurtherEvtes (p_blk_len) ; // skip Presentation Block 

} 

} // Reach Meta Data 

FIG. 13 illustrates an example presentation of a media missile in a player 

according to specific embodiments of the present invention. In a further embodiment, a 

player can use a local ads. htm/ page by default, instead of always needing to contact a 

server for the html page. The local ads. html can include references to other files, such as 

avi, gif, jpeg, html or similar files. In specific embodiments, these may be restricted to 

the same director}^ in which ads. html resides and its sub-director^^ AdsJitml and related 

files can be packaged into a media missile format, with, as an example, a name in the 

form 'playcr.dat'. 



EXAMPLE SYNCMROiMZED AUTHORING MANAGER IMPLEMENTATION 

An SAM according to specific embodiments of the invention provides a user- 
friendly GUI that allow'S an author to successfully complete a Media Missile (or, in some 
embodiments, just a well-formed Presentation Specification) by applying mouse clicks 
and keystrokes, with no programming background required. SAM provides a wizard- 
driven guide to help non-programming users generate multimedia presentations. SAM 
wizards generate MM based on a series of templates, so that the multimedia presentations 
created through SAM are well-formed and can be parsed and rendered by any compliant 
player. In further embodiments, SAM provides a tool to preview different media such as 
text, images, himl files, audio and video clips and facilitates the generation of 
synchronization data between different media. SAM can be implemented in Java or other 
programming environments. FIG. 14 illustrates an example general method for guiding a 
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user in creating a media missile or presentation data according to specific embodiments of 
the invention. 

Template Support 

A SAM according to the present invention provides a set of different standard 
5 templates to allow authors to create presentations such as video lectures, headline news, 
and slide shows. 

For example, a video lecture template makes it easy for an author to generate a 
video lecture (video synchronized with HTML slides). 

A headline news template is designed for virtual briefings (video synchronized 
10 with text streams at bottom) in scenes such as: headline news — video with anchorperson 
and text containing headlines; online event — video with event introduction and text 
containing registration information; product advertisement — video with product 
advertisement and text containing pricing infonnation. 

A slide show template is designed for illustrated audio (audio synchronized with 
15 images) in scenes such as: guided tour ~ narrated map with scenic photos to guide .users 
through a site tour; process presentation — audio commentary to slides demonstration of a 
process; or online publication -- sound book or magazine with reference materials in 
pictures. 

While some prior multimedia authoring tools supported only non-streamed data 
20 types, an SAM according to the present invention, allows inclusion of streaming media 
along with other media types. FIG. 15 illustrates an example of a Synchronized Authoring 
Manager (SAM), also refetred to as a Synchronized Authoring Tool (SAT), according to 
specific embodiments of the invention. 

SAM Core Components 

25 In specitlc embodiments, SAM includes four components: Presentation Wizard, 

Preview Manager, Presentation Specification Data Manager, and Media Missile 
Generator. 

Presentation Wizard is SAM's kernel that provides an easy way to enhance 
disparate media into integrated and dynamic multimedia content. It guides users to create 
30 presentations in just a few steps. 

Preview Manager can display any allowed medium element. If the medium is 
time-irrelevant, such as an image, a text stream, or an html file, then the exact content 
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will be shown in the preview window. If the medium is time-intrinsic, such as an audio or 
video clip, then it will be played continuously from the beginning in the preview window. 
If the medium is from a live streaming source, such as a live camera or video feed, a live 
~ sampre"of IRc liiedium'cairbe'pray windoxv; "The Pre'vievv "Manager can 

be activated inside and outside the Presentation Wizard. Users can use the Preview 
N4anager to preview the medium before incorporating it into a multimedia presentation. 

Presentation Specification Data Manager is responsible for creating 
Presentation Specification Data (referred to at times in this context as metadata, although 
herein it refers to data for the presentation specification and not the Media Missile 
MetaData described above) information and a presentation specification, which is used to 
synchronize different media within a multimedia presentation. Users can playback an 
audio or video clip or stream, meanwhile recording relevant timing information for static 
media based on the content of the clips. For example, a user wishing to create a 
presentation that displays a narrated map fusing audio), showing some slides of scenic 
photos (using images), and access from a live outdoor camera, indicating current weather. 
With Presentation Specification Data Manager, the user can decide the timing point to 
show the next slide according to the audio content or to open a window to display the 
streaming (or live) data. The information is saved in a data structure and used later for 
creating the corresponding presentation content. 

Document Generator features automatic creation of a presentation specification 
and/or a Media Missile. The document's structure is based on one of the templates, and 
the media and timing information come from Presentation Wizard operation. The 
Document Generator can package the various media data with the presentation data to 
create and any necessary or executable content to create a Media Missile. As an option, 
the Document Generator can also output just an SMIL-type presentation specification. 

Creating a Media Missile Using SAM 

Step 1 ■ Choosing A Template 

At first, users choose the type of presentation they wish to create. Each template is 

designed with a specific purpose in mind. SAM can support templates such as: 
I idea Lecture Template 

Video lecture usually consists of a video clip or a streaming feed from a streaming 
data source such as a live camera or continuously encoded television broadcast, 
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synchronized with HTN4L slides. The video can be any media format supported by a 
target Player. The HTML slides are any confomiing HTML files or other files that can be 
handled by a player. In some embodiments, if the slides are originally in another format 
(such as Microsoft PowerPoint) they are first converted to HTML format using a ''Save as 
5 HTML" command in PowerPoint. 

Headline Nen^s Template 

Headline news usually consists of a video clip or a streaming feed synchronized 
with text streams. The video clip can be any media format supported by a target player. 
Text streams are generally formatted ASCII text. In a specific embodiment, users can 
10 directly type any text they w^ant in the second step. 

Slide Show Template 

Slide show usually consists of an audio clip or streaming audio feed synchronized 
with image slides. The audio clip can be any media format supported by a target Player. 
The image slides are image files in GIF, JPEG, or any supponed format. 

15 Step 2 - Adding and Previewing Media (Preview Manager) 

The second step is to add media for the presentation. Users can activate the^Open 

File dialog box to select the files or streaming sources they wish to add. Users can also 
activate the Preview Manager to preview^ any selected medium. In various situations, 
users can decide how to create presentation data by using either the Presentation 

20 Specification Data Manager or SAM's default setting. If the Presentation Specification 
Data Manager is chosen, then the wizard will activate the Presentation Specification Data 
Manager as the third step. If SAM's default setting is chosen, then SAM assumes, for 
example, that slides are displayed sequentially, each slide has the same duration and the 
first slide starts displaying at the very beginning of the presentation. Therefore, SAM can 

25 calculate the presentation specification Presentation Specification Data automatically and 
does not need to activate the Presentation Specification Data Manager. 

Step 3 - Creating Presentation Specification Data (Presentation Specification 
Data Manager) 

In this step, the Presentation Specification Data Manger is activated to create the 
30 presentation specification Presentation Specification Data. Presentation specification 
Presentation Specification Data is used to synchronize different media within a 
multimedia presentation. It is as simple as timing infonnation w^hen to change to the next 
slide (image, text, or html file) or when to open a window for live media stream, possibly 
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synchronized with the content of time-intrinsic media (such as video or audio clips). With 
Presentation Specification Data Manager, users can decide at which timing point a slide 
or window starts displaying based on the timeline of another playing audio or video clip. 
An~exampie^implementation-of-a- video-lecture,- in~whiclv two slide-files- (formatted-as— 
HTML files) and a opening of a live streaming source are synchronized to an explanatory 
video clip based on the Presentation Specification Data file shown below: 
rsldOO I. htm>123 1646556 

I sld 002.htm>54643123r3 2Z 

: livesource.s21>7634537334 

In the Presentation Specification Data file, the string before character is the 

indicator or handle of one slide file or a streaming source, while the string after character 

is the exact media time when the slide or streaming source should start displaying. 

Whenever the current media time exceeds one of the media times in the Presentation 

Specification Data file, the corresponding displayed is activated. Since the media times 

are associated with the timeline of one video clip, then the synchronization between the 

video and the slide files is implemented. 

Step 4 - Generating Media Missile Presentation (Document Generator) 

Finally, the wizard shows the summary information such as the template name of 
the presentation, file names or URLs of selected media, and the Presentation 
Specification Data for synchronization. After clicking Finish button, the Presentation 
Wizard sends all the infomiation to a Document Generator. The Document Generator 
creates a SMIL presentation specification or MM document based on a selected template 
file. In one embodiment, it reads the template file line by line, parsing it, and collecting 
and then placing the corresponding media data, tile names, URLs or presentation data 
into a Media Missile structure. 

SAM Component and Screen Implementation Examples 

FIG. 16 illustrates an example of a Preview Manager GUI according to specific 

embodiments of the present invention. This Preview Manager is a single window, split 
into a media preview area and a control area at the top of the window. A text field control 
contains the current medium file location. Users can choose a medium in any way know 
for selecting files or indicating locations or streaming sources (URLs) such as typing a 
new path inside the text field; or pressing the button labeled ... to select a location. The 
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selected path is then shown in the text field. After selection, users can press Preview 
button to start displaying the medium- 
According to specific embodiments, each Presentation Wizard step can be 
implemented as a dialog box. Each dialog box has three buttons in common. Users can 

5 click Back button to return to the previous wizard step or click Next button to go forward 
to the next wizard step. Users can click Cancel button to close the wizard at any step 
without generating a document. 

FIG. 17A-C illustrate examples of choosing various presentation templates 
according to specific embodiments of the present invention. 

10 The Adding and Previewing Media functionality can change based on the 

template selected. FIG. 18A shows the Video Lecture template Add and Preview Media 
screen. Labels such as "Add Video Clip'\ "Add Slide Files'', "Video Duration^', and 
"Each Slide Duration" are set based on the media support of this template. Users can 
add, remove, or preview media in this dialog box. Users can click Remove button to 

15 delete selected slide file in the list box. Users can also click Preview button to activate the 
Preview Manager for playing back the selected medium file. Figure 7-19 shows a selected 
HTML slides displayed in the Preview Manager. 

FIG. 19A illustrates the situation when Headline News template is selected in step 
one. Headline News consists of video synchronized with text notations, which come from 

20 direct input instead of text file. Therefore, a dialog box FIG. 19B is provided for direct 
text input and is activated when Add Text Notations button is clicked. 

Based on the template selection, GUIs can also present different Presentation 
Specification Data creation methods. FIG. 19 A, for example, shows "Using Presentation 
Specification Data Manager" radio button, which means the wizard will activate the 

25 Presentation Specification Data Manager dialog box as the third step. "Using Default 
Setting'' radio button indicates that users just need to type in the duration data. The 
wizard assumes that each slide has the same duration and the first slide starts displaying 
at the very begmning of the presentation. Therefore, the wizard does not need to activate 
the Presentation Specificafion Data Manager and goes directly to the last step - step 4. 

30 Similarly to Step 2, the funcfionality of the Presentation Specification Data 

Manager changes based on the template selection in step one. FIG. 20A shows an 
example when users select the Video Lecture template in step one and choose some 
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media files in step two. The ''Video Lecture" label is shown in the Presentation 
Specification Data Manager. Labels such as Video File", "HTML Slide Lisf\ "HTML 
Slide Preview", and "HTML Slide File Name" are set based on the media support of 
Video Lecture template. The left part of the Presentation SpecificaFion Data Manager is 
5 the video monitor area. The file path is shown in the Video File text field, and the video is 
rendered and played by a Preview module. When the video starts playing, the 
MediaProgressMonilor activated in step two updates the video Current Time every 50 
milliseconds. 

The right part of the dialog box is the slide timeline setting area. Each slide is not 
10 only included in the HTML Slide List box, but also shown in the Slide Timeline 
Recording panel in order. Users can click Reset button to reset each TimeLine text field 
in the panel to 0.0 second and click Sort TimeLine button to sort the slides in timeline 
order. FIG. 20B shows the situation when users select the Headline News template in step 
one. 

15 As shown in Figure FIG. 20A, Slide Timeline Recording panel in Presentation 

Specitlcation Data Manager is the GUI part dynamically generated according to the 
users' selection in step one and two. Each slide file corresponds to one small child panel 
that consists of a File Name text field, a Record button, and a TimeLine text field. Since 
the video timeline is the entire video lecture presentation timeline, users can click Record 

20 button to set the starting time of the corresponding slide based on the video content. 
When one Record button is clicked, the exact time is shown in the corresponding 
TimeLine text field and the following slide is shown in the HTML Slide Preview area. 

As a further option, according to specific embodiments, when Users click the Sort 
button, the slides in the Slide Timeline Recording Panel are sorted in timeline order, 

25 which means the slide with the earliest starting time shows in the first line of the panel 
and so on. Bubble sort is simple and effective; therefore, it can be used as a sort algorithm 
for the timeline. 

Because generating presentation is the last step in the wizard guide, presentation 
information such as template name, selected media files, automatically generated or 
30 manually generated (by Presentation Specification Data Manager) Presentation 
Specification Data is summarized in the Presentation Information text area. Meanwhile 
users can define the presentation title, author name, and copyright. When Finish button is 
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clicked, the Document Generator will get all the infomiation and generate a presentation 
independently. FIG. 21 illustrates an example last step of a presentation generation. 

SAM can utilize live multimedia streams by using Real-time Transport Protocol 
(RTP) for streaming media objects, and Real-time Transport Control Protocol (RTCP) for 
5 controlling and managing remote stream objects like a traditional VCR does (play, stop, 
and pause). RTP is the Internet standard for the transport of real-time data including 
audio and video stream. RTP can be used for media-on-demand application as well as 
interactive services such as Internet telephony. RTCP is the control protocol designed to 
work in conjunction with RTP. 

10 Further Example Source File 

Below is one example of a SMIL-type presentation specification file that could be 

pan of a media missile according to specific embodiments of the present invention arid 
could be accessed through a synchronized media player according to specific 
embodiments of the present invention. To display this file on a target machine, for 
15 example, would require players such as video, audio, HTML text, and image players, 
each of which could be resident at a target machine or packaged in a media missile. 

<smi 1 > 
<head> 

<meta name= " author " cont:ent: = " S . Shim" /> 
20 <meta name= " copyright " content =" SJSU " /> 

<iayout > 

<root - layout background- color = "192,192,192- v/idth= " 12 00 " 

height="620"/> 

<region id= " videoregion" left="40" top="10" width^"150" 
25 height="100" z-index="0" background- 

color="lS2, i92,192"/> 
< region id=" lectureregion " lef t = " 10" T:op="22 0 " width="4 50 " 
height ="2 00" z-index="0" background- 

color="192, 192, 192 "/> 
30 <region id= " s 1 ideregion " left="240" top="10" width="250" 

height="200" z-index="0" background- 

colGr="192, 192,192"/> 

< / layout > 
</head> 
35 <body> 
<par> 

<video src= "demo\gm . mpg" region^ " videoregion " / > 
<seq> 

<html src= "demo\carl .gif " begins "5s" dur="3s" 

40 region= " slideregion " / > 

<html src= "demo\car2 .gif" dur="5s" region^ " s lideregion" / > 
<html src= "demo\car3 .gif" dur="5s" region= " si ideregion" / > 
<html src= "demo\car6 . gif " dur= " 5s region^ "slideregion" / > 
<html src= " demo\car5 .gif" dur= " 5s " region= " si ideregion" / > 

45 <html src= "demo\car4 . gif " dur="5s" region= " si ideregion" / > 

<html src= "demo\car5 . gif " dur="5s" region^ " s 1 ideregion "/ > 
</seq> 
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<seq> 

<html src="demo\gml .html" begin="6s" dur="6s" 

region= " lectureregion" / > 
<html src="demo\gm2 .html" dur="6s" region= " lectureregion" / > 
5 <html src="demo\gm3 .html" dur="6s" region^ " lectureregion" / > 

<html src="demo\gm4 .html'' dur="6s" region^ " lectureregion" / > 

^ html~s r c^"ciemd"\"gm5 rh t mT"— du r^"'6's "" "^r eg i on^"Te"c tur e re'g i~on ""/-> 

</seq> 
</par ^ 
10 </body> 
</smil> 

Embodiment in a Programmed Information Appliance 

FIG. 22 is a block diagram showing a representative example logic device in 

which aspects of the present invention may be embodied. The invention can be 

15 implemented in hardware and/or software. In some embodiments of the invention, 
different aspects of the invention can be implemented in either client-side logic or a 
server-side logic. As will be understood in the art, the invention or components thereof 
may be embodied in a fixed media program component containing logic instructions 
and/or data that when loaded into an appropriately configured computing device cause 

20 that device to perform according to the invention. As will be understood in the art, a 
fixed media program may be delivered to a user on a fixed media for loading in a users 
computer or a fixed media program can reside on a remote server that a user accesses 
through a communication medium in order to download a program component. 

FIG. 22 shows an infonnation appliance (or digital device) 700 that may be 

25 understood as a logical apparatus that can read instructions from media 717 and/or 
network port 719. Apparatus 700 can thereafter use those instructions to direct ser\^er or 
client logic, as understood in the art, to embody aspects of the invention. One type of 
logical apparatus that may embody the invention is a computer system as illustrated in 
700, containing CPU 707, optional input devices 709 and 711, disk drives 715 and 

30 optional monitor 705. Fixed media 717 may be used to program such a system and may 
represent a disk-type optical or magnetic media, magnetic tape, solid state memor>^ etc.. 
The invention may be embodied in whole or in part as software recorded on this fixed 
media. Communication port 719 may also be used to initially receive instructions thai are 
used to program such a system and may represent any type of communication connection. 

35 The invention also may be embodied in whole or in part within the circuitr>^ of an 

application specific integrated circuit (ASIC) or a programmable logic device (PLD). In 
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such a case, the invention may be embodied in a computer understandable descriptor 
language which may be used to create an ASIC or PLD that operates as herein described. 

Other Embodiments 

The invention has now been described with reference to specific embodiments. 

Other embodiments will be apparent to those of skill in the art. In particular, a user digital 
infomiation appliance has generally been illustrated as a personal computer. However, the 
digital computing device is meant to be any device for interacting with a remote data 
application, and could include such devices as a digitally enabled television, cell phone, 
personal digital assistant, etc. 

While the forgoing and attached are illustrative of various aspects/embodiments of 
the present invention, the disclosure of specific sequence/steps and the inclusion of 
specifics with regard to broader methods and systems are not intended to limit the scope 
of the invention which finds itself in the various permutations of the features disclosed 
and described herein as conveyed to one of skill in the an. 

Furthermore, while the invention has in some instances been described in terms of 
client/server application environments, this is not intended to limit the invention to only 
those logic environments described as client/server. As used herein, ''client*' is intended 
to be understood broadly to comprise any logic used to access data from a remote system 
and ''server" is intended to be understood broadly to comprise any logic used to provide 
data to a remote system. 

It is understood that the examples and embodiments described herein are for 
illustrative purposes only and that various modifications or changes in light thereof will 
be suggested by the teachings herein to persons skilled in the art and are to be included 
within the spirit and purview of this application and scope of the claims. 
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WHAT IS CLAIMED: 

1 . A nielhod of accessing streaming media content comprising: 
initiating a media missile; 

executing a scheduler, said scheduler loading a presentation specification; 
under control of said scheduler, initiating one or more players; 

passing indications of media objects in said media missile to one or more players; and 
at said one or more players, under control of said scheduler, presenting said media 
objects from said media missile. 

2. The method of claim I further comprising: 

at said one or more players, retrieving one or more headers from said media missile 

corresponding to a media object indicated for said one or more players; 
at said one or more players, inspecting said one or more headers to determine the 

necessity of downloading lightweight software components from said media missile 

to render the media data; and 
at said one or more players, loading media-specific components at runtime if needed to 

access said media data. 

3. The method of claim i wherein said accessing is performed on a mobile platform 
with limited pre-installed software capability. 

4. The method of claim 1 wherein said accessing is performed on a mobile platform 
with limited memory footprint. 

5. The method of claim 1 wherein said accessing enables synchronized presentation 
on a mobile platform. 

6. The method of claim 1 further comprising: 

at said one or more players, setting one or more barrier synchronization primitives. 

7. The method of claim 1 wherein said one or more players plays media data 
according to an enable signal from said scheduler. 
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8. The method of claim 1 wherein said initiating comprises launching a media 
missile from local storage, said media missile including a initial player scheduler able to 
run on a local device's runtime platform. 

9. The method of claim 1 wherein said initiating comprises launching a media 
5 missile from remote storage, said media missile including a initial player able to run on a 

local device's runtime platform. 

10. The method of claim 1 wherein said initiating comprises accessing a presentation 
formatted in a web page. 

1 1. The method of claim 1 wherein said scheduler is taken from said media missile. 
10 12. The method of claim 1 wherein said scheduler is preinstalled on an access device, 

13. The method of claim 1 wherein said presentation specification is loaded from said 
media missile. 

14. The method of claim 1 wherein one or more of said indications of media objects 
includes a signature. 

15 15. The method of claim 1 wherein said media missile is a document container 
including active content and passive content, said active content comprising machine- 
aware executable components and said passive content comprising presentation 
specification, metadata information, and media data. 

16. The method of claim 1 wherein said media data can include another media 
20 missile. 

17. The method of claim 15 wherein said active and passive content of said media 
missile can be accessed from a single streaming file location. 

1 8. A method of accessing streaming media content comprising: 
initiating a media missile; 

25 wherein said media missile is a document container capable of holding machine-aware 

active content and passive content in a variety of formats; and 
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passing indications of media objects in said media missile to one or more players. 

19. The method of claim 18 wherein said initiating comprises: 
_ using a b rowser to point-and-shoot sajd^niedia miss^lg^ 

20. The method of claim IS wherein said initiating comprises; 

using a command line interface to launch a local self-contained media missile. 

21 . The method of claim IS wherein said initiating comprises: 
initiating a player with media missile name as input. 

22. The method of claim IS wherein said initiating comprises: 

wherein said media missile contains a complete executable component and is able to 
load itself and lam in accordance with runtime loader characteristics available on a 
particular operating system platfonn. 

23. The method of claim IS wherein at least one of said one or more players is 
included in said media missile. 

24. The method of claim 18 wherein at least one of said one or more players is an 
external player. 

25. The method of claim 1 8 wherein said external player is an MPEG player. 

26. The method of claim 18 wherein a hyperlink retrieval of said media missile in a 
web-enabled page implicitly streams only an initial active machine-aware portion of said 
media missile with the remainder of said media missile streamed under control of said 
initial active machine-aware portion. 

27. The method of claim 18 wherein said accessing may be performed on a platfonn 
with limited pre-installed software capability. 

28. The method of claim 18 wherein said accessing is perfonned on a platform with 
limited memor\^ footprint. 
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29. The method of claim 18 wherein playback of a Media Missile further comprises 
potentially overlapping processes of delivering, assembling, and rendering media missiles 
and their contents. 

30. A method of playing a synchronized multimedia presentation comprising: 
initiating a media missile; 

wherein said media missile is a document container capable of holding machine-aware 

active content and passive content in a variety of formats; 
initializing a synchronized multimedia player with a timeline-based scheduler to 

schedule playback according to a presentation specification; and 
passing indications of media objects in said media missile to one or more players. 

31. The method of claim 30 wherein said presentation specification is available 
through a web page. 

32. The method of claim 30 wherein said presentation specification is available from 
said media missile. 

33. The method of claim 30 wherein said synchronized multimedia player is resident 
on a platform prior to said initiating. 

34. The method of claim 30 wherein said synchronized multimedia player is in said 
active content of said media missile. 

35. The method of claim 30 further comprising: 

parsing said presentation specification to build a presentation tree; and 
firing said playing from the top of said tree based on the synchronization 
characteristics of intermediate tree nodes. 

36. The method of claim 35 further comprising: 

for one or more parallel nodes, creating independent execution threads to handle each 
parallel task, each independent thread referring to a global time base. 

37. The method of claim 35 further comprising: 
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for one or more sequential nodes, using a single execution thread to step through 
sequential tasks. 

3^8^_The method oF claim 35 Further comprisinR; 

upon firing a leaF node, locating a correct media player and media data. 

39. The method of claim 30 further comprising: 

if a link to media data in said presentation specification is a relative path, searching 

said media missile for said media data; and 
if a link to media data in the specification is not a relative path, following an absolute 

path to locate the media data. 

40. The method of claim 39 wherein said absolute path is either specified entirely in 
said link or is initiated from a root of a ser\^er from where said media missile was 
streamed. 

41. A method of accessing a multimedia presentation comprising: 

checking a media missile to detemiine if said media missile contains an executable 

portion for a current platform; 
if said checking is positive, launching machine-aware components in said media 

missile on the fly or as an enhancement of local software to render media data 

within said media missile; and 
iFsaid checking is negative, launching a local media player with default components to 

read media data from said media niissile: wherein said media missile is a document 

container capable of holding machine-aware active content and passive content in a 

variety of Formats. 

42. The method of claim 4 1 funher comprising: 

initializing a synchronized multimedia player with a timeline-based scheduler to 

schedule playback according to a presentation specification; and 
passing indications of media objects in said media missile to one or more players. 

43. The method of claim 42 wherein said player handles both barrier and event 
synchronization with elastic timers. 
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44. The method of claim 42 wherein said player internally handles synchronization 
events according to a presentation specification. 

45. The method of claim 44 wherein a presentation specification can define precise 
timing sequence or a relaxed schedule. 

46. The method of claim 44 wherein a playback can be either sequential or concurrent 
(parallel). 

47. A media missile streaming document container comprising; 
a first location for active media content; and 

a second location for passive media content; 

wherein said active media content comprises machine-aware executable code and said 
passive media content can comprise presentation specification, metadata 
information, and media data. 

48. A streamable media missile file comprising: 
a first portion for holding executable content; 

a second portion for holding a presentation specification; 

a third portion for holding metadata; and 

a fourth portion for holding one or more media data objects. 

49. The device according to claim 47 wherein one or more of said media data objects 
can include another media missile, providing a nested format. 

50. The device according to claim 47 wherein said media missile file can be managed 
at a server and distributed from a server as a single file. 

51. The device according to claim 47 wherein said media missile file is generated on 
the fly at a serv^er though is delivered to a user as a single streaming file. 

52. The device according to claim 47 further comprising: 

a first length field indicating if machine-executable code is present and the length of 
said code if present; and 
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a first continuation field indicating presence of other portions in said media missile 
file. 

5 3. The dev i ce ac cording to cl aim 5 2 wherein: 

if said first continuation field is logical true and said first length field indicates a zero 

length, said media missile begins with said presentation specification; 
if said first cominuation field is logical false and said first length field indicates a zero 

length, said media missile begins with said metadata; and 
if said first continuafion field is logical false and said first length field indicates a 

negative value, said media missile comprises pure media data. 

54. The device according to claim 47 further comprising: 

a second length field indicating if a presentation specification is present and the length 

of said presentation specification if present; and 
a second continuafion field indicaUng presence of other portions in said media missile 

file. 

55. The device according to claim 54 wherein: 

if said second continuation bit field is logical true, then metadata information follows 

said presentation specification; and 
if said second confinuation bit field is logical false, then media data follows said 

presentation specification. 

56. The device according to claim 54 further comprising; 

a third length field indicating if metadata is present and the length of said metadata if 
present; and 

a third continuation field indicating whether data after said metadata is another media 
missile or media data. 

57. The device according to claim 56 wherein: 

said metadata comprises one or more media object length fields indicating length of 
one or more media data objects in said media missile. 

58. The device according to claim 57 wherein: 
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wherein said one or more media objects are stored in sequence according to a sequence 
of said media data length field(s). 

59. The device according to claim 56 wherein one or more media data length fields 
includes a field indicating whether a corresponding object is a resource locator or is a 
payload. 

60. A method of assisting a user in creating a synchronized multimedia presentation 
comprising: 

prompting said user to select a template for said presentation; 
prompting said user to select one or more media data objects; 

prompting said user to select an option for generating presentation specification data; 
and 

preparing final output, including a presentation specification. 

61. The method according to claim 60 wherein said final output comprises a 
multimedia missile that can include media objects, a presentation specification, metadata, 
and active components in a container deliverable as a stream from a single streaming 
source location. 

62. The method according to claim 60 wherein said final output comprises a 
presentation specification in a standard descriptive tagged language. 

63. The method according to claim 60 wherein said method enforces generation of a 
well-formed synchronized multimedia presentation. 

64. The method according to claim 60 wherein final output can include both precise 
and elastic synchronization primitives. 

65. The method according to claim 60 further comprising: 

prompting said user to modify duration settings for media data objects; and 
prompting said user to use a presentation metadata manager to customize a 
presentation timeline. 

66. The method according to claim 65 further comprising: 
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displaying the video and corresponding slides at a particular point to an user to assist 
creating a presentation. 

67. Th e method acco rding to claim 6 1 furt her comprising: 

prompting said user to select whether to keep the final output locally or upload it to a 

specific hosting ser\^er. 

68, The method according to claim 60 further comprising: 

accepting presentation specification and media data as input to said authoring 
method. 

69, The method according to claun 60 further comprising: 

creating a top-down tree representation of synchronization primitives wherein each 
intermediate node is either a parallel or sequential synchronization primitive; and 

determining required fields according to a media missile specification for tenninal leaf 
nodes in said tree. 

70. The method according to claim 69 further comprising: 

attaching fields to individual media data specified in a presentation so that said 
individual media have the format of a media missile with media data only. 

7 1 . The method according to claim 70 further comprising: 

using a depth-first tree traversal to calculate and create optional fields based on user 
input; and 

using a breadth-first tree traversal to traverse the entire tree and create the final media 
missile. 

72, A system for streaming synchronized multimedia presentations comprising: 
a real-time data source providing a phirality of streaming media objects; 

a data store for holding and retrieving presentation specifications or media missiles; 
and 

a repository manager that transmit a streaming media missile to a target device, where 
the media missile can include packaged indications from said real-time data source, 
can include static media objects, and can include presentation specification, 
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73. The system of claim 72 further comprising: 

an authoring manager for receiving authoring specifications for a streaming 
presentation. 

74. The system of claim 72 wherein said authoring manager accepts indications 
5 provided by an author and using said indications prepares a well-formed streaming 

presentation in accordance with a predefined descriptive tagged language. 

75. The system of claim 72 wherein said repository manager handles transmission of 
executable components as said components are needed and are requested by a 
synchronized multimedia player said for accessing a portion of said streaming 

10 presentation. 

76. The system of claim 75 w^herein said repository manager and said player are able 
to besin pre-fetching delay-sensitive media data. 

77. The system of claim 72 wherein data is delivered from said repository manager in 
a container that provides indications for machine-aware executable code, media data, and 

15 associated information. 

78. The system of claim 77 whereni said container can include one or more additional 
containers in a nested structure. 

79. The system of claim 77 wherein said container holds all active and passive content 
necessary to play a synchronized multimedia presentation. 

20 80. The system of claim 72 wherein said authormg manager understands various 
language specifications, fonnats, and templates in synchronized multimedia. 

81. The system of claim 72 wherein said authoring manager includes a wizard that 
guides a user to compose a synchronized presentation and retrieve media data from a 
media repository manager to produce a media missile. 

25 82. The system of claim 72 further comprising a synchronized multimedia player 
including an internal scheduler to process delay-sensitive data. 
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83. The system of claim 82 wherein said synchronized multimedia player can present 
presentations in accordance with SMIL presentation data. 
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Example METADATA block FORMAT 

// in this example always 0; 



// 
{ 



d a t a__o r _m i s s i 1 e 

block is data 
metaDataBlkLen ; 
signature ; 

// f ileFormat Id 
payload_or_url ; 
def aultUrlLen; 
bitl payload_or_url ; 
int6 3 f ilenameLen ; 
bitl payload_or_url ; 
int63 f ileDataLen; 
} f ileNameDataLenStruct [numberOf Files] 
mediaDataBlock 

char defaultUrl [def aultUrlLen] ; 

{ 



next 



int63 
intl28 

bitl 
int63 

{ 



// sizeof {metaDataBlock) ; 
// 0x03010101 => player version 3 

version 1 
// always 1; URL 

// always 0/ => payload 

// always 0; => payload 



01. 



char filename [f ilenameLen] ; 
char fileContent [f ileDataLen] ; 
} fileNameDataStruct [numberOf File] ; 
} mediaDataBlock; 
// ending 
bitl p_blk_exist; 
int6 3 eLength; } 
// be aware the byte order difference in file vs in memory 
// e.g. int value READ ( ) ; File Hex Data "12 34 56 78"; 
// value in Hex is 0x78563412; 

FIG. 9 



wo 01/26378 



7/17 



PCT/USOO/27634 



EXAMPLE METADATA BLOCK CONTENTS 
Assumed values 



strlen 20 



Default URL is " http : / / v^v^/v.' . abcCo . com" 
c:\webAds\s21WebAds.htm nameLen 13, size 2562 bytes 

main page 

c : \webAds\otherLinks . htm nameLen 
c : \webAds\logo.gif nameLen 
c : \webAds\ sponsor \abcCo . jpg nameLen 



/ / the 



14 , 
6 . 
17 , 



size 1548 bytes 
size 5874 bytes 
size 18745 bytes 



Mefcada 

bitl data_or_missile 
int63 metaDataBlkLen; 
intl28 signature; 
bitl payload_or_url ; 
int6 3 def aultUrlLen; 
bitl payload_or__uri ; 
int63 f ilenameLenl ; 
bitl payload_or_url ; 
int63 f ileDataLenl ; 
bitl payload_or_url ; 
int63 f ilenameLen2 ; 
bitl payload_or_url 
int63 f ileDataLen2 ; 
bitl payload_or_url ; 
int63 f ilenameLen3 ; 
bitl payload_or_url ; 
int63 f ileDataLen3 ; 
bitl payload_or_url ; 
int63 fil enameLen4 ; 
bit.l payload_or_url ; 
int63 f ileDataLen4 ; 
// At 96 bytes; this is the 
char defaultUrl [20] 
char f ilenamel [13} 

encryption) 
char f ileDatal [2562] 
char f ilename2 [14] 

encryption) 
char f ileData2 [2562] 
char filenames [8] 
char f ileDataS [2562] 

encryption) 
char f i lename4 [17] 
char f ileData4 [2562] 

encryption) 
bitl cont ination_bit 
int63 eLength // 0 



fca Media Missile values 

// 0 

// (64/8) * (4+4*2) = 96 bytes 
// 0x03010101 
1 
20 
0 
13 
0 

2562 
0 
14 
0 

1548 
0 
8 
0 

5874 
0 
17 
0 



// 
// 
// 
// 
// 
// 
// 
// 
// 
// 
// 
// 
// 
// 
// 
// 
// 

// 18745 
media data 

// "http://www.abcCo.com" 

// "s21WebAds.htm" (simple 

// s21WebAds .htm' s data 

// "otherLinJcs.htm" (simple 

// otherLinks . htm' s data 
// "logo.gif" 

// logo.gif s data (simple 



/ / " sponsor \abcCo . j pg " 
// sponsor\abcCo . j pg ' s 

// 0 

FIG. 10 



data (simple 
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MEDIA MISSILE BLOCK CONTENTS EXAMPLE 

char executablecode [nnnnnn] ; // Executable code 
bitl p_spec_cont:inat ion_bi t ; // 1 
int63 pbength; 

-eha-r p-r e s en-tat- i-on--spec-[pppp J-;- -/-/- 



// PPPP 
Rre s en t a t-i on- 



S p e c i f i c a t i on- 



■ da t^a j_o r _m i s s i 1 e 


II 


0 


met'aDat aBlkLen ; 


1 1 


( 64/8 ) 


signature ; 




// Ox 


payload_or_url ; 


II 


1 


def aultUrlLen; 


II 


20 


payload_or_url ; 


II 


0 


f ilenameLenl ; 


II 


13 


payload_or_url ; 


1 1 


0 


f i leDataLenl ; 


II 


2562 


payload_or__url ; 


II 


0 


f ilenameLen2 ; 


II 


14 


payload_or_url ; 


II 


0 


f ileDataLen2 ; 


II 


1548 


payload__or_url ; 


■II 


0 


f ilenameLen3 ; 


II 


8 


payload_or_url ; 


II 


0 


f ileDataLen3 ; 


II 


5874 


pay load_or__url ; 


II 


0 


f ilenameLen4 ; 


II 


17 


payload_or_url ; . 


II 


0 


f ileDataLen4 ; 


II 


18745 



(4 + 4"*2) ^ 96 bytes 



bitl 
int63 
intl28 
bitl 
int63 
bitl 
int63 
bitl 
int63 
bitl 
int63 
bitl 
int63 
bitl 
int63 
bitl 
int63 
bitl 
int 63 
bitl 
int63 
// 
// 

//At eLength + pLength plus 104 bytes; this is the media data; or note 
that 

// if the data_or_missile bit = 1, 
// media missile. In that case, 
above will 
have the form: 

bitl payload_or_url ; // 
int63 MediaMissileLenl ; // 
and the media data will include 
char mediamissile [mmmm] 



// 
// 
// 
// 
// 
// 
// 

char 
char 



char 
char 

char . 

char 

char 

char 
char 

bitl 
int63 



this portion could begin another 
the words after the signature block 



0 

the object 



defaultUrl [20] 


II 


"http : 11^-^ . abcCo , com" 


f ilenamel [13] 


II 


"s21WebAds.htm" (simple 


encryption) 






f ileDatal [2562] 


II 


s2 IWebAds . htm ' s data 


f ilename2 [14] 


II 


"otherLinks.htm" (simple 


encryption) 






f ileData2 [1548] 


II 


otherLinks . htm ' s data 


f ilename3 [8] 


II 


"logo.gif" 


f ileData3 [5874] 


II 


logo.gif 's data (simple 


encryption) 






f ilename4 [17] 


II 


" sponsor\abcCo . j pg " 


f ileData4 [18745] 


II 


sponsor \abcCo . j pg • s dat, 


encryption) 






continuat ion_bit 


II 
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eLength // nnnnnn 
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B1 



prompt a user to select a presentation 
- template 


i 


82 


prompt a user to select one or more media 
data objects 


i 


B3 


prompt a user to select an option for 
generating presentation specification data 


i 


B4 


preparing final output data including a 
presentation specification 


i 


85 



if indicated by user, packaged final output 
data, including media objects and indications 
of media objects, presentation specification, 
metadata, and active components into a 

media missile container streamable from a 



single location 
FIG. 14 
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Media Missile File 
or Specification 




FIG. 15 



.^Visual Cafe - PrevieWWindow.VEP 



-ta|x| 



£ile £dtt Yiew Search ErojeO Insert Layout Qbjeci Qotobase lools "^^^^^ tl Q'P 



"D tS^ B : ^ tfefe;^"- ^ >t?^ Workspa ce: |Ed.t zl -^^^ ^^^^^^ 



^ AVvT| AWT DntaBQund | AWT Additions | AWT Multimedia | Swng | icebrowsgrbean | Swing Containers Dot; 



• £f PreviewManager : 
!indB;n 
- previewBtn 
locationTF 
previewBeanl 



S Form Pesigrter - PrevicwManaqer 




For Help, press F1 



iNUM 



FIG. 16 Preview Manager GUI 
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^SAT Presentation Wizard — Choose Template 



This Pf Rseiitatloi* WizartJ cjuide you ihf oiiyh the process of credtimj a rnunimetlia presentation 



Laryout useii in template 



Template: 



I Video Lecture 



These templates inlciale itreilefined track types 
ami " layouts settintj. 



jplay a video wtth slides. 

'Suggested application. 
Video Presentation. 



N8Xt> 



FIG. 17A Choosing a Video Lecture Template 



Cancel 



SAT Presentation Wizard — Choose Template 



1 his Presentation Wizard n"»ie you ttirouijh the process of creatinij a inuttlmeiJia presentation 
SMIL Layotrt used in template I etrtplate: 



Headline News 



These templates tnicude prcdet'inerl tracK tyjies 
and layouts setttnij. 



Play a video with text streams. 

iSuggesied application, 
i News Broadcasting. 



Next> 



Cancel 



FIG. 17B Choosing a Headline News Template 
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jSAT Presentation Wizard — Choose Template 



I his Piesentation Wizard (lutde you (hroiajh the process uf creatitm a 



tmmitttctlid presenldtton 



Layout used in tetnplaie 



Tfimpltite: 




I Slide Show 



These templaies inlcude prccJctiDeil Kdck tyjies 
atid layoins senituj. 



iPiay an image wiln a soundtrack. 

iSuggested apoiication: 
Family vacation slideshow. 
product demo, 

corporate training presentation 



Next> 



Cancel 



FIG. 17C Choosing a Slide Show Template 
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SAT Presentation Wizard — Add Media 



Video Lecture 
Add video Clip: 



:F:\nnastetldance.avi 



Add Slide Files: 



;F:\mastertdoctdance\sldOOl.htm 
iF\mastertdoc\dance\sld002.mni 



Preview 



Add... 



Remove 



Set I irne for Medi^: 

Using Metadata Manager 

] Using Default Setting 

Video fjuratiort: 
Eacti Slide Uui dtion: 



El 



seconds 
seconds 



Preview 



<BacK I Next> Cancel 



FIG. 18A Video Lecture Add And Preview Media 
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SAT Preview Manager 



j F :\m a s ten d'o'cVd a n ce Vs t d 0 0 1 Thtrrr 



Preview- 



Dance - Slide One 
•romantic 

breath-taking 




Close 



FIG. 18B Video Lecture Template Preview Selected Slides 



I SAT Presentation Wizard — Add Media 



Headline News 

Add Video Clip: 



■El 



iF:\master\video,avi 



Preview 



Add Text Notations: 



jFirst Jump!aBeaut!f'jl TurniQWoo! 
itnto the water..acool.,.a8ravo!!i 



Add... 



Set rune lor Media: 

Using Metadata Manager 

Cj Using DefauR Settinp 

\/lUeo (juration: 
Each Text Duration: 



seconds 
seconds 



Preview 



<Back Ne)ct> i ; Car^cel 



FIG. 19A Headline News Template - Add And Preview Media 
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Add Text Notation for Headline News 



Add Text Notation for Headline News: 



First Jump! 
Beautiful Turn! 
Wo oil 



OK 



FIG. 19B Add Text Notation For Headline News 



3 SAT Presentation Wizard — Generate Metadata 



Video Lecture 

\riiteo Hite: .F:Vn a stertdance.avi 




Cm rent Time/ Total Tiint;: .1 8.CU1 8.0 



HTML SlifJe List: HTML Slitie Preview: 

I F:\master\doc\dancel.. ^\ SlitlC 1 \VO>^ 



Reset 



mding 



Sort Time Line 




HTML Slide File Name 



F:\mastertdoc\dance\sld001.hl Record ! ;0.0 



F:lmastertdoc\dance\std002.h 



Record 1 5.3 



<Back 



N8Xt> 



Cancel 



FIG. 20A Video Lecture Template - Create Presentation Specification 

Data 
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I SAT Presentation Wizard — Generate Metadata 



Headline News 

-V/iueo.Pile:^- . FjXmastertvideo.ayJ 



[X] 



fe xt_No t ancjtt lis l : 





JJ 



Current Time/ Total Fiine: 1 4.6/20.0 



I First JumpiaBeautiful , ^ \ 



Reset 



Text Notdiioii Preview: 



Into the water.. 
Coot... 
Bravo!!! 



Sort Time Line 



: -irstJump! 
' 3eautiful Turn! 
j ^oo! 

I Into the water.. 
I Cool... 
i Bravoll! 



Record ; 3.3 



Record i14,4 



<Back 



Next> 



Cancel 



FIG. 20B Headline News Template - Create Presentation Specification 

Data 



SSAT Presentation Wizard — Finish 



You have cornpleteti usiny ttie Presentation Wizard. Your rmittirnetfia presentation will now t)e createtl. 



El 



Presentation Information: 



Video Lecture: 

Video File; F:\master\dance.avi 
Video Duration; 1 e.Osec 

Metadata manually generated 

First HTML Slide File Begin at O.Osec 

HTML Slide File and Duration; 
F:Xmaster\doc\dance\sldOOl htm 1 5.8sec 
F:\master\docWancetsid002.htm 2 "! 999998sec 



,-.-,-:-:v-:.v:yv:s.-.-.-1 



Presentation Title: 



jvideoLecturei 



Airtlmr Name: 



lYing Wang 



CofTyriyht: 



!SJSU 



<Back Finish 



Cancel 



FIG. 21 Video Lecture Template - Generate Document 
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